Turn the tables, ask them REAL questions if they confront you about not going to the meetings. It doesn't even have to be about the cult, it can be about you - "I've got some personal things to work out" then ask "Is there something NEW at the KH that I haven't heard before?", "Are the FDS teaching real life-skills? Like how to get a decent job so I can afford to live - because that's what I really need right about now!".
You are only responsible for your own feelings. Try and figure out why you feel the way you do about this situation. Why are you willing to give in to pressure from people who don't care enough about you as an individual to leave you at peace with your own decision, let alone take care of you?
Determine your own values - things that are important to you and why, then learn to articulate those values and practice it whenever you can. Part of having healthy self esteem is speaking up for yourself. You can do it with tact or not. It's up to you. If you have the choice of them being uncomfortable or you, which will you choose and for what reasons?
I don't think there's a right or wrong in this - you can go along or not, either way it will have an affect on you. The choices that you make will determine if the affects are positive or negative from your pespective.
sweet tee
You have a choice in everything and every situation. You can choose to allow people to manipulate you and make you feel bad, or you can choose not to let other peoples feelings override your own. Forget what's going on with everyone else and get to know and understand who you are. That's priceless!